Confidence Estimation for Automatic Speech Recognition Hypotheses

Automatic speech recognition (ASR) systems produce transcriptions for audio which sometimes contain errors. It is useful to know how much condence may be placed in this output being correct. Condence estimation is concerned with obtaining scores which quantify this level of condence. e development and application of a principled, žexible framework using conditional random eld (CRF) models for condence estimation is described. Errors tend to occur over a number of consecutive words in ASR output. is phenomenon is not typically accounted for in condence estimation, but is exploited here through the sequential nature of the CRF. A custom CRF framework is developed, making it possible for useful feature functions to be engineered. is framework is extended to support hidden-state CRFs. To inform this condence estimation model, novel predictor features indicative of the quality of ASR hypotheses are proposed, along with a technique for their extraction from lattices. e CRF-based approach is used to combine multiple predictor features and estimate condence scores for words in ASR hypotheses. is yields performance improvements in the normalised cross entropy (NCE) metric of up to 11.4% relative to a strong baseline (using decision trees). e novel application of a hidden-state CRF to this task yields further relative improvements of up to 17.2%. Estimating condence scores on the sub-word-level is also investigated. Sub-word-level features are combined with word-level features to yield improvements of up to 31.7%relative. e use of a hiddenstate CRF for this task yields even larger relative gains of up to 48.6%. e application of CRFs to estimate keyterm condence scores for spoken term detection is proposed. Discriminative features for keyterm hypotheses are introduced, as well as a model-based approach to keyterm score normalisation. is approach results in improvements of 26% and 36% relative in the miss rate and false alarm rate at operating points of interest. e novel task of detecting deletions within ASR output is investigated. e sequential nature of the CRF is exploited to make this possible, such that regions in which deletions occur are modelled. Modelling word condence and deletion regions simultaneously yields an approach which is capable of detecting deletions. Overall, the proposed framework for condence estimation is shown to yield improved condence estimates. is is important for downstream applications (e.g. dialogue systems, keyterm detection) which make decisions based on these scores, as well as in-system applications (e.g. data selection and adaptation).